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SUMMARY 


Visual displays drive the human operator’s highest bandwidth sensory input channel. Thus, 
no telemanipulation system is adequate which does not make extensive use of visual displays. 
Although an important use of visual displays is the presentation of a televised image of the work 
scene, this paper will concentrate on visual displays for presentation of nonvisual information 
(forces and torques) for simulation and planning, and for management and control of the large 
numbers of subsystems which make up a modern telemanipulation system. 


INTRODUCTION 


Teleoperation consists of the control of a remote manipulator in order to perform mechanical 
actions usually associated with the function of the human arm and hand. This extension of manual 
dexterity to hostile environments requires high sensory feedback bandwidth to replicate perceptual 
inputs normally available to the human. 

Augmented by computers and advances in robot sensor development, the application of teleop- 
eration has been extended to the areas of deep sea, underground, and space exploration. Future 
space missions will require a more advanced teleoperator with automation capability to perform 
many new tasks including satellite retrieval or repair, space station construction, and payload 
handling (ref. 1). 

Visual displays drive the human operator's highest-capacity input channel, allowing an impor- 
tant means of closing the dextrous manipulation loop. The televised image of the work scene 
affords the operator an important means of receiving qualitative and nonsymbolic quantitative 
information about the work environment. This type of display has the advantage of providing 
information in a natural, unencoded form, but can suffer from perspective ambiguities if any 
parameters such as the viewing angle, lighting conditions, display resolution, refresh rate, or 
reference frame are ill chosen (refs. 2 and 3). Additionally, televised display can rapidly exhaust 
the available transmission data rates in the downlink. Displays which represent the state variables 
in encoded form offer a much more efficient use of the downlink if their chosen form can be 
quickly decoded and easily understood by the human operator. 

There are many important parameters to be displayed in telerobotic displays. Displays can 
provide information about the proximity of the end effector to goals and obstacles (ref. 4); the 
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forces and moments exerted at the wrist frame of the manipulator (ref. 5); the current configuration 
and work envelope of the manipulator relative to objects in the task space, including regions near 
manipulator singularities that should be avoided; and mass distribution of objects in the 
environment. 

This research was performed at the Jet Propulsion Laboratory, California Institute of Technol- 
ogy, under contract with the National Aeronautics and Space Administration. 


Force Torque Displays 

A long-term effort in our laboratory has focused on the display of forces and torques arising 
from a remote manipulator's interaction with the environment. Visual displays complement the 
ability of force-feedback master manipulators when time delay, or control-station constraints pre- 
clude such aids. We have developed and evaluated several graphical formats through which this 
nonvisual task space information can be presented including horizontal bar graphs (ref. 5), so- 
called "star diagrams" (ref. 6), and various enhancements such as color coding, event-driven flags, 
and true perspective presentation. The star diagram display has recently been tested in over 21 hr 
of experimental teleoperation with resulting guidelines for future system design. 


Simulation 

An Iris graphics workstation has served as a graphics engine for a number of simulation dis- 
plays used for kinematic analysis of proposed telerobotic task scenarios. Examples include an 
animated simulation of a dual-arm, satellite- servicing task and a detailed simulation used for analy- 
sis of arm-base location and position in a dual-arm teleoperation laboratory. These perspective 
displays can be interactively rotated and zoomed in and out to give three-dimensional information 
to the operators without the problems of stereo displays. Visual enhancements such as color cod- 
ing, reference grids, and manipulator work volume projections are used in place of binocular cues. 


Executive Control Displays 

In a full telerobotic system, a very large number of subsystems and capabilities need to be con- 
trolled. In full systems, these will include two arms, hand controllers, and smart hands; trading 
and sharing of control between autonomous and telerobotic modes; and control of cameras, light 
sources, and other sensory systems. The traditional solution of large racks of subsystem control 
panels attended by a dedicated operator can be improved upon with an executive workstation which 
can communicate with all subsystems over a local network such as Ethernet. Recent exploration 
work has developed a prototype display architecture based on the desktop metaphor built into 
workstations such as the Macintosh or Sun. Icons representing each of the subsystems to be 
controlled populate a workstation screen representing the control domain. The key feature is that 
an operator can selectively attend to one of a large number of subsystems by selecting (clicking) its 
icon. The icon expands into a software control panel which displays the subsystems 1 status and 
accepts commands. Alarm conditions can be indicated on the icon to alert attention to the particular 
subsystem. 
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Telemanipulation displays are not limited to on-line situations during task execution. They can 
provide predictive information about the outcome of given operator-control actions if the manipu- 
lator and environmental characteristics are modeled (refs. 7 and 8), as well as a play-back for 
postmortem analysis of operator performance. Off-line use of all previous modes with the addition 
of environmental modeling allows for training of operators in routine activities with a minimum 
investment in hardware and low risk of damage to training facilities. The success of this approach 
can be seen in the widespread acceptance of flight simulators as a training tool for commercial and 
military pilots. Currently, one validated, high-fidelity, real-time simulator for space telemanipula- 
tion exists, the Shuttle remote manipulator simulator at the Johnson Space Flight Center (ref. 9). 

This paper reports the results of display research and development at the Jet Propulsion Labo- 
ratory Advanced Teleoperator Development Laboratory in three sections: displays of force and 
torque data; perspective projection displays of simulated manipulators and task environments; and 
executive control of complex telemanipulation systems by direct manipulation. 


Force/Torque Information 

When a robot manipulator interacts with the environment, forces and torques are exerted at the 
contact points. Information from the load cells in the robot "wrist" can be resolved into three ^ 
forces and three torques representing the interaction between manipulator and environment as felt 
in the wrist. The specific coordinate system for this resolved information is arbitrary, but a useful 
one is to resolve the three components of force along the x, y, and z axes, and the three compo- 
nents of torque to the pitch (x), yaw (y), and roll (z) axes of the manipulator hand. 

Although considerable attention has been focused on using backdrivable master manipulators to 
provide contact force information to the operator, there are cases where this direct information is 
insufficient or impractical. In particular, visual displays complement the ability of force-feedback 
master manipulators when time delay, numerical accuracy, or control-station constraints preclude 
such aids. Graphical displays can also indicate task-specific constraints which must be satisfied 
during manipulation and whether the constraints are met. Our laboratory has developed and evalu- 
ated several graphical formats through which this inherently nonvisual, but spatial, information can 
be presented (fig. 1). 

The most basic format, developed first, is a set of horizontal bar graphs in which each of the 
six forces and torques is displayed (fig. la) (ref. 5). This type of display has been tested in our 
laboratory (ref. 10) and in the simulated Space Shuttle cargo bay at the Johnson Space Flight Cen- 
ter (ref. 9) where it has been shown to reduce the magnitude and duration of forces required to 
complete a task. However, the horizontal bar graph display fails to represent the spatial content of 
the force/torque information because the assignment of the forces and torques to the bars is essen- 
tially arbitrary. 

In the JPL-OMV (Orbiting Maneuvering Vehicle) smart hand (ref. 6), an improved display was 
developed which represented a primitive perspective view of the unit vectors making up die hand 
reference frame (fig. lb). Torques were represented by bar graphs crossing the appropriate axes. 
This type of display was tested in over 21 hr of experimentally recorded teleoperation with opera- 
tors of various experience levels performing simulated satellite servicing tasks (ref. 11). 
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In these experiments, the JPL-OMV smart hand was mounted on the Prototype Flight Manipu- 
lator Arm (PFMA) at Marshall Space Flight Center, and operators performed task-board operations 
from a remote control room. The operators were provided with three camera views of the scene, a 
six-axis "joyball" controller (ref. 12), and the previously described star display of forces and 
torques. Task performance was measured in terms of RMS forces/torques required to perform the 
task. Low RMS forces/torques indicated the absence of forcing and jamming of the tool and thus 
better task performance. Force and torque display was available to the operators in selected trials 
and comparative force control performance was measured for the two cases. Although operators 
reported that the visual force/torque information was useful, no significant reduction was observed 
in task-related forces and torques. 

When an earlier version of this display was tested on the space shuttle RMS simulator 
(ref. 13), reductions in forces were demonstrated. This discrepancy can be attributed to the rela- 
tively poor position-control performance of the PFMA and its high stiffness, versus the highly 
accurate position control capability of the RMS and its low stiffness. In the absence of true force- 
control capability, operators apparently adopt a strategy of controlling forces by commanding small 
position increments against the stiffness of the manipulator and load. 

This type of indirect control strategy demonstrates that in the case of telemanipulation, it is very 
difficult to evaluate displays in isolation— especially in terms of overall task performance. 

A further display refinement is to generate a three-dimensional bar graph in which the magni- 
tude of each force component is displayed in the direction of its unit vector. Torques are displayed 
as circular bar graphs centered on the axes. This display has been rendered in color and true per- 
spective on an IRIS graphics workstation (fig. lc). Evaluation in use awaits integration of the 
IRIS with actual telerobotic hardware. 


Event-Driven Displays 

We are currently developing enhancements to improve these force/torque displays. In many 
tasks, the desired outcome is to perform a manipulation subject to specific constraints. For exam- 
ple, the task may be to press on a latch such that z-axis force is greater than or equal to 10 lb and x 
and y forces and all torques are less than 1 lb (or ft-lb). The burden of checking these constraints 
can be removed from the operator by a set of display primitives which indicate the constraints on 
each axis and a global flag indicating that all constraints are satisfied. These "event-driven" dis- 
plays also have served to combine information from proximity, tactile, and force/torque sensors 
(refs. 14 and 15). ' ’ 

This concept has been tested with a light-emitting diode (LED) version of the star pattern dis- 
play at Johnson Space Center (ref. 5) and has been added to the OMV smart-hand display. 

One key issue is the value of detailed visual force/torque information to the operator relative to 
other visual information sources, especially cameras. Future experimentation will address this 
question by forcing the operators to choose among display sources and recording relative fre- 
quency of selection of each display. A cost will be imposed for switching between sources to pre- 
vent the adoption of a scanning strategy. Thus, for example, operators will attempt to minimize a 
time score for completing a manipulation task, but will be penalized 1 sec each time they switch 
from the various displays. 
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In present force/torque displays, the operator must transform the display from the hand frame 
to the static frame for the display. The manipulator control device is usually referenced to task 
space and the operator can be assumed to map the various camera views to a mental model of the 
task space. Knowledge of the position and orientation of the manipulator end effector in task space 
is required to perform this mapping. Incorporating this mapping into a task-space display presents 
the technical issue of interfacing the hand electronics to the manipulator control system (to provide 
task space position and orientation), and the design issue of how to present the task-space infor- 
mation. Alternatives being considered are to transform the star display to the end-effector position 
and orientation, transform it again to the viewplane of one of the cameras, and superimpose it on 
the camera view. Another possibility is to create a synthetic deformable object such as a striped 
cylinder, locate it at the manipulator wrist, and deform it according to the forces and torques pres- 
ent at the wrist. A display of the deformed cylinder superimposed on the video scene would give 
an easy-to-grasp, intuitive picture of the manipulator's interaction with its environment. 


Real-Time Perspective Simulation 

Simulation presents an effective means of developing teleoperator systems, can provide valu- 
able feedback during the use of such a system, and can be an effective design tool. 

In our laboratory setup, a universal 6 degree-of-freedom, force-reflecting hand controller 
(FRHC) is used as master and a PUMA 560 robot is used as slave. The kinematics and dynamics 
of both arms are extensively studied and described in the literature (refs. 16 and 17). Two 
National Semiconductor NS-32016 microprocessors were chosen to control the FRHC and the 
PUMA arm, respectively. The distributed control and interface information is detailed in refer- 
ence 18. A real-time simulator also was built in parallel with the distributed control system to 
facilitate human control performance studies, hardware/software checkout, and operator training. 

The real-time simulator (fig. 2) consists of almost all the hardware of the complete telemanip- 
ulation system except that the PUMA manipulator is replaced by the computer graphic simulation. 
The 6 degree-of-freedom FRHC is the key interface between the operator and the control station. 

It provides the necessary force feedback to the operator and is equipped with six optical encoders 
for position sensing and six motors for backdriving the operator. The control-station processor 
interprets the encoder values and converts them into joint angles. It then performs forward kine- 
matics calculations to determine the end position of the FRHC in the work space and then transmits 
those position commands to the remote station. The remote processor receives the position com- 
mand from the control station, computes inverse kinematics of the PUMA arm, and determines the 
desired joint angles which are sent to the graphics processor for animation. The Silicon Graphics 
IRIS work station is employed for the graphics generation and display. It animates the movements 
of the PUMA arm in color graphics and provides the task-simulation environments. 

The requirements for the display are that animation be generated at a rate high enough that the 
simulation appears continuous and realistic to the operator. The Silicon Graphics IRIS 2400 is a 
UNIX-based graphics workstation which uses a highly pipelined display architecture. The 
IRIS 2400 contains several VLSI hardware graphics processors known as geometry engines 
(ref. 19). These are capable of performing basic graphics operations, such as matrix transforma- 
tions, clipping and mapping to device coordinates at a rate of approximately 65,000 three- 
dimensional, floating-point coordinates per second. The geometry engines are arrayed to form the 
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graphics pipeline, with a 68000 microprocessor used as a low-level pipeline manager. The host 
processor for the geometry pipeline is a 68010 which communicates with it over the multibus. The 
geometry engines are accessible via the C graphics library provided with the system. This library 
enabled high-level operations such as coordinate frame and polygon definitions to be specified 
from within the Applications Program. 

The PUMA-560 model was created into two steps. The constituent graphical objects such as 
the links and base were defined relative to their own coordinate frames. Appropriate coordinate 
frames were then developed for each link in a fashion similar to the Denavit-Hartenberg link speci- 
fications. Links were subsequently displayed in their appropriate coordinate frames, thus forming 
the complete model of the robot. The necessary link parameters were found in Craig (ref. 17). 

The frame transformations for the forward kinematics of the PUMA were inherent to the 
graphical model of the PUMA. Robot motion animation was achieved by varying the appropriate 
link parameters, i.e., the joint angles, and rapidly redrawing the robot model according to these 
new values. 

The capability to perform high-speed graphics computations permitted the display of a model of 
intermediate complexity at approximately a 10-Hz refresh rate, including input/output operations. 
Data are sent from the hand controller to the IRIS in 16-bit binary form over the RS232 serial 
interface. 

Hidden-surface elimination was investigated, but not implemented in this version of the simu- 
lation display because of speed constraints. Several fast software algorithms for hidden-surface 
elimination exist. The general principle involved is to presort the polygons composing a static 
object before they are displayed. Unfortunately, while these techniques work well for a roving 
viewpoint and static objects, the links in the PUMA model are constantly changing their position 
relative to each other and thus their constituent polygons are not presortable. 

There are many applications for the graphics simulation of the PUMA 560 running on the IRIS 
workstation. Of most value is its use as a debugging tool. Many software modules developed for 
control of the manipulator can be tested and debugged using the graphics simulation without 
actually using the manipulator. In general, the simulation allows its users to test-control software 
when the actual manipulator is not available, or its design is not yet finalized. Different manipula- 
tor geometries can be explored for functionality before they are actually prototyped. This flexibility 
is true for hand-controller design as well. 

The simulation also can be used as a tool for training teleoperator system operators. Fictitious 
objects can be introduced into the virtual work environment so that operators can practice pick-and- 
place tasks as well as more complex operations without endangering hardware. Using the IRIS 
system’s ability to clip against an arbitrary plane, end-effector collision with objects in the virtual 
environment could be detected and indicated in real time. This feature will assist the operator in 
practicing teleoperation and collision avoidance. 

When a significant time delay in communication exists between the controller and manipulator 
(e.g.. Earth-based control station commanding a geosynchronous satellite servicing teleoperator) a 
graphic simulation could become valuable in enhancing operator performance. By overlaying a 
stereoscopic wire- frame view of the manipulator on the stereoscopic television image of the task 
space, a predictive display can be obtained (ref. 20). This allows the operator to immediately 
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realize the ramifications of his/her actions before actually performing an operation. Commands 
could be buffered and then sent once the operator is sure that no damage will result from given 
actions. 


Teleoperator Laboratory Design Simulation 

We have also used perspective displays as a design tool to explore various layouts for a dual- 
arm telemanipulation laboratory. These simulations (fig. 3a) allowed the designers to specify robot 
base location and posture (elbow up/down, shoulder in/out, etc.) in a model of the actual labora- 
tory space. A grid placed on the floor represents the actual floor tiles so that the simulation can be 
easily related to the actual space. A projection of the maximum extent of each robot's work volume 
was drawn on the floor grid. The intersection of the two work-volume projections gives an idea of 
the cooperative work volume of the two robots. Note that the work volume is a function of arm 
configuration if arm flips are not allowed. Another important design issue direcdy addressed by 
this display is the visibility of the task space and especially the manipulator end effectors by the 
operator (in direct operation from the control station) or from a particular camera. Because the 
viewpoint of the simulation can be changed dynamically, designers can view the robots from any 
contemplated control station or camera mount. On the basis of this simulation, the plan shown in 
figure 3 was shown to have higher cooperative work volume and better sight lines from the 
operator control station than a competing plan. 


Simulated Satellite Servicing Animation 


Autonomous task-sequence simulation takes the static scene simulation a step further by adding 
the element of time and order of subtask execution. Our application is an animation of two robots 
performing the replacement of an attitude-control system on the Solar Max Satellite (fig. 3b). This 
is the chosen scenario for the 1988 telerobot demonstrator project at JPL. The simulation is adapt- 
able to a variety of tasks, and could take input from artificial intelligence task planners to provide a 
means of human verification of the output of autonomous subsystems. 


Executive Control Displays 


A complete telemanipulation system requires far more interaction with operators than that 
required for the purely manipulation components of a task. Considerable human interaction over- 
head will be required to control cameras, select system operating mode, attend to error conditions, 
start up and shut down the system, and hand off control between teleoperation and automatic oper- 
ation Many of today's telemanipulation systems require a second operator and control station to 
perform these "executive" functions. The nature of this task is to selectively attend to details of 
whichever one of a large number of subsystems requires attention. 


Desktop Control Station 

The traditional approach to this executive control station is a console or series of racks filled 
with a separate control panel for each subsystem. An attractive alternative is offered by a single 
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control station consisting of a large bit-mapped display through which an operator can control all of 
these functions. 

We are currently prototyping such an executive-control display which compresses all 
executive-control functions into a single high-resolution workstation screen (fig. 4). Control 
interaction will take place between the workstation and the subsystems over a local area network. 

The basis for this display is the desktop direct-manipulation environment (refs. 21, 22, 
and 23) as implemented in the Macintosh and the Sun workstation, and pioneered in the Smalltalk 
environment (ref. 24), which evolved from earlier work such as Sutherland’s Sketchpad system 
(ref. 25). The workstation screen represents a domain which is populated with icons representing 
the various systems. The operator can expand a subsystem icon to reveal a complete control panel 
for that system containing buttons, indicators, sliders, graphics displays, and so forth. Icons can 
be dynamic so that alarm conditions can reach the attention of the operator when a subsystem is 
closed. 

We have prototyped examples of icons from such a system on a Sun workstation (at the dis- 
play and human-interaction level only). A control station based on this concept will take up much 
less space than a conventional panel rack and will be very flexible with respect to future expansion. 
Operators could easily customize the display to the requirements of a specific task. 

Interaction between the manipulation operator and the icon-based executive control station is 
desirable, eliminating the need for a second operator even in two-handed teleoperation. In cur- 
rently planned dual-arm teleoperation systems, the operator's hands are occupied with controlling 
two slave manipulators through six-axis, force-reflecting, hand controllers. Either hand controller 
(depending on operator preference) could be temporarily changed over to controlling a display cur- 
sor on the executive-control station. 


Hand Controller as Mouse 

In this concept, the operator will press a button mounted on the hand controller, which will 
lock the slave manipulator, or turn its control over to an automatic or intelligent control system. 
Two degrees of freedom of the hand controller would then control the location of the cursor on the 
executive control display. The hand-controller backdnve capability could be used for providing 
detents indicating cursor position relative to the icons. This will provide an active assist in moving 
the cursor to small icons or panel objects. Designation of display objects (analogous to clicking a 
mouse button) will be accomplished by the hand-controller button normally used for gripper con- 
trol. Other hand-controller degrees of freedom could be used to operate panel items such as knobs. 

For example, to adjust an analog quantity such as a rate limit, the operator could move the hand 
controller and thus a screen cursor to a picture of a knob representing the appropriate quantity. The 
location of the cursor on the screen will be taken from the x and y coordinates of the hand con- 
troller. In the immediate neighborhood of the "knob," the operator will feel a small force generated 
by the control computer to represent the negative gradient of a small "potential function" on the 
workstation surface. The potential function contains "wells" around each of the icons and panel 
items. This force will guide the operator to the icon and correct small positioning errors. When 
the cursor is over the knob, the roll axis of the hand controller would be used to change its setting. 
Other types of icons would be operated by orthogonal hand-controller motions. For example, a 
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toggle-switch icon would be operated by the pitch axis. This provides a measure of safety because 
each icon can be activated only by a particular hand motion. The icons should be designed and 
linked to the axes of hand motion so the way to actuate them is intuitive. 

At the conclusion of the executive control function, the operator would resynchronize the slave 
manipulator with the master and resume manipulation. The details of the transitions between 
manipulation control and cursor control are complex, but identical in principle to those used in the 
indexing function already designed into such systems. 
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(a) A simple set of horizontal bar graphs, one bar for each of the six axes of force and torque. 


Figure 1- Displays of force/torque information for telerobotics. Several formats have been devel- 
oped and experimentally evaluated at JPL for the display of forces and torques encountered by 
a remote manipulator to the controlling operator. Panels (a) and (b) represent monochrome 
displays, (c) represents color. 
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(b) A pseudo-perspective display in which the bar graphs are aligned with unit vectors representing 
the direction of action of forces, and the roll, pitch, and yaw axes for torques. 

Figure 1.- Continued. 
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c) A true perspective, full-color display. 
Figure 1.- Concluded. 
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Figure 2 - Real-time simulation of a robot manipulator in telemanipulation. The wire-frame simu- 
lation substitutes for the manipulator for software validation or operator training. The complete 
telemanipulation system consists of a hand controller (left); control processors (not shown), 
monochrome display; and optionally, robot manipulator (background). The display computer 
is plug-compatible with the manipulator controller. 
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Figure 4 - Executive-control display for a telemanipulation system. Icons representing each of the 
many subsystems involved in a full, dual-arm, telemanipulation system are displayed on a sin- 
gle monochrome workstation screen. Subsystems are controlled through a pointing device 
operating simulated switches, sliders, and buttons. In conventional systems, these functions 
are controlled by a second operator sitting at a large rack of hardware control panels. The 
executive control display can eliminate the need for a second operator because the manipulation 
operator can operate the display using the force-reflecting hand controller. 
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